Planning and Managing the Development of Complex Software Systems

I n the initial decades of the computer industry, systems were built in isolation. Large problems typically were solved through the development of numerous, stand-alone, vertical solutions. As computer systems started covering progressively larger segments of the problem space, they began to overlap and often failed to integrate. The solution to this problem was part systems engineering, and part grand design and grand implementation. The premise behind these grand approaches was to think of the entire problem and to implement a total solution. Although on the surface this seems perfectly reasonable, the problem is that this strategy, especially when coupled with a Waterfall Model life-cycle development approach, simply does not reliably scale up to implementing large, complex systems. Depending on whose data you reference , there is anywhere from a 50 percent to an 80 percent likelihood that any given software system project will fail [1]. That is, the project will require substantially more time than originally planned, cost substantially more than originally budgeted , or will deliver substantially less func-tionality than originally expected, or any combination of the preceding. Furthermore , the larger the project, or the longer the planned duration, the greater the likelihood of failure. After nearly 25 years in the software industry and 10 years of conducting process appraisals using the Software Engineering Institute's Capability Maturity Model ® for Software, I have repeatedly observed that it is exceedingly rare for complex, large-scale, multi-year, grand implementation software projects to deliver all expected functionality within the originally planned schedule and budget. But are these failures truly due to the grand implementation approach, or is it something else? In this article, grand implementation approaches are considered to be coupled with a Waterfall Model life-cycle development approach. Although large, complex systems often need a grand design to ensure overall architectural integrity, having such a design does not mean that the next step must be a grand implementation. Nevertheless, after a grand design is completed development often commences for the entire system. When the entire system is ready, it is put though integration test, system test, and acceptance testing: classic Waterfall Model development. While this can be a very effective approach for small systems or short duration projects, it becomes a much less successful approach as project duration and complexity increase. Part of the problem is requirements volatility. The traditional response to this problem is to require the customer to freeze …